How to restore from Amazon ES Manual Snapshot

How to restore from Amazon ES Manual Snapshot

#オペレーション
#Amazon Elasticsearch Service
#AWS
yusuke

yusuke

facebook logohatena logotwitter logo
Clock Icon2018.08.05

この記事は公開されてから1年以上経過しています。情報が古い可能性がありますので、ご注意ください。

Amazon ES has both Automated Snapshots and Manual Snapshots, but an Automated Snapshot cannot be used to restore to a new Amazon ES domain.　Therefore, if we want to restore to a new Amazon ES domain, we will need to restore from a Manual Snapshot.

In this post, we will introduce how to take a Manual Snapshot and how to restore from the Manual Snapshot.

Step1: Create S3 bucket

We create an S3 bucket to store the Manual Snapshot. We create it using the name es-test-index-repo via the S3 console.

We write down the S3 bucket ARN as it will be used later.

  • arn:aws:s3:::es-test-index-repo

Step2: Set up IAM policy

We use the IAM console to create the policy es-test-index-repo-policy from Policies -> CREATE policy as shown below.

In the Resource section of the policy, we specify ARN arn:aws:s3:::es-test-index-repo to refer to the S3 bucket we created earlier.

{
   "Version":"2012-10-17",
   "Statement":[
       {
           "Action":[
               "s3:ListBucket"
           ],
           "Effect":"Allow",
           "Resource":[
               "arn:aws:s3:::es-test-index-repo"
           ]
       },
       {
           "Action":[
               "s3:GetObject",
               "s3:PutObject",
               "s3:DeleteObject"
           ],
           "Effect":"Allow",
           "Resource":[
               "arn:aws:s3:::es-test-index-repo/*"
           ]
       }
   ]
}

We then create the IAM Role es-test-index-repo-role from Roles -> Create role in the IAM console.

We attach the policy created earlier and set the trust relationship as follows.

Trust Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

We also create an attach the following policy to the users who will be creating the Manual Snapshot repository.

  • es-backup-policy
{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "iam:PassRole",
        "Resource": "arn:aws:iam::aws_account_id:role/es-test-index-repo-role"
    }
}

Step3: Register Manual Snapshot repository

Before taking a Manual Snapshot for the first time, we must register the Manual Snapshot repository.

It is necessary to sign the AWS request here. curl does not support AWS request signature.

There is sample code for Python, which we will refer as reference.

register-repo.py

  • AWS_ACCESS_KEY_ID = ''
  • AWS_SECRET_ACCESS_KEY = ''
  • host = ''　# ES domain endpoint
  • region = 'us-west-2'
  • path = '' # Name of the snapshot repository
  • bucket　= '' # Name of the S3 bucket
  • role_arn = '' # The ANR of the IAM role created earlier
import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID =''
AWS_SECRET_ACCESS_KEY =''
region = 'us-west-2'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo'
url = host + path

payload = {
  "type": "s3",
  "settings": {
    "bucket": "s3-bucket-name",
    "region": "us-west-2",
    "role_arn": "arn:aws:iam::aws_account_id:role/es-test-index-repo-role"
  }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.text)

Execute

$ python register-repo.py
{"acknowledged":true}

Step4: Take a Manual Snapshot

If the access policy specifies an IAM user or role, we must sign the snapshot request. Since there is sample code in the documentation, we will use it as reference.

If you need a signature request, please refer to the Python code below; if you do not need a signature request, please refer to the curl example below.

snapshot.py

  • path # Specify the snapshot name
import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-2'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo/my-snapshot-1'
url = host + path

r = requests.put(url, auth=awsauth)

print(r.text)

Execute

$ python snapshot.py
{"accepted":true}

curl

$ curl -XPUT 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/my-snapshot-2'
{"accepted":true}

Step5: Check snapshot repository

check_repository.py

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-2'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/?pretty'
url = host + path

r = requests.get(url, auth=awsauth)

print(r.text)

execute

$ python repository.py
{
  "es-test-index-repo" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "es-test-index-repo",
      "region" : "us-west-2",
      "role_arn" : "arn:aws:iam::aws_account_id:role/es-test-index-repo"
    }
  },
  "cs-automated" : {
    "type" : "s3"
  }
}

curl

$ curl -XGET 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot?pretty'
{
  "es-test-index-repo" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "es-test-index-repo",
      "region" : "us-west-2",
      "role_arn" : "arn:aws:iam::aws_account_id:role/es-test-index-repo-role"
    }
  }
}

Step6: Confirm snapshot in repository

check_snapshot.py

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-2'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo/_all?pretty'
url = host + path

r = requests.get(url, auth=awsauth)

print(r.text)

Execute

$ python check_snapshot.py
{
  "snapshots" : [ {
    "snapshot" : "my-snapshot-1",
    "uuid" : "**********aixSzBQFI_2g",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [ "index1", "index2" ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T02:12:52.086Z",
    "start_time_in_millis" : 1527214372086,
    "end_time" : "2018-05-25T02:12:55.745Z",
    "end_time_in_millis" : 1527214375745,
    "duration_in_millis" : 3659,
    "failures" : [ ],
    "shards" : {
      "total" : 10,
      "failed" : 0,
      "successful" : 10
    }
  }, {
    "snapshot" : "my-snapshot-2",
    "uuid" : "**********Svf55AYG0QuQ",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [ "index1", "index2" ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T02:13:36.289Z",
    "start_time_in_millis" : 1527214416289,
    "end_time" : "2018-05-25T02:13:38.326Z",
    "end_time_in_millis" : 1527214418326,
    "duration_in_millis" : 2037,
    "failures" : [ ],
    "shards" : {
      "total" : 10,
      "failed" : 0,
      "successful" : 10
    }
  } ]
}

curl

curl -XGET 'elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/_all?pretty'
{
  "snapshots" : [ {
    "snapshot" : "my-snapshot-1",
    "uuid" : "**********aixSzBQFI_2g",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [ "index1", "index2" ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T02:12:52.086Z",
    "start_time_in_millis" : 1527214372086,
    "end_time" : "2018-05-25T02:12:55.745Z",
    "end_time_in_millis" : 1527214375745,
    "duration_in_millis" : 3659,
    "failures" : [ ],
    "shards" : {
      "total" : 10,
      "failed" : 0,
      "successful" : 10
    }
  }, {
    "snapshot" : "my-snapshot-2",
    "uuid" : "**********Svf55AYG0QuQ",
    "version_id" : 6020299,
    "version" : "6.2.2",
    "indices" : [ "index1", "index2" ],
    "include_global_state" : true,
    "state" : "SUCCESS",
    "start_time" : "2018-05-25T02:13:36.289Z",
    "start_time_in_millis" : 1527214416289,
    "end_time" : "2018-05-25T02:13:38.326Z",
    "end_time_in_millis" : 1527214418326,
    "duration_in_millis" : 2037,
    "failures" : [ ],
    "shards" : {
      "total" : 10,
      "failed" : 0,
      "successful" : 10
    }
  } ]
}

Step7: Restore from snapshot

We will try two patterns here.

  • Restore to same ES domain
  • Restore to other ES domain

Step7-1 Restore to same ES domain

If there is an index with the same name, the restore will fail so we need to first delete it. ※ Amazon ES does not support Elasticsearch _close API.

delete_index.py

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-1'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

# DELETE INDEX

path = 'my-index'
url = host + path

r = requests.delete(url, auth=awsauth)

print(r.text)

$ python delete_index.py
{"acknowledged":true}

curl

Delete all indices

curl -XDELETE'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_all'

Delete specific index

$ curl -XDELETE 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/index-name'

restore-all.py

The path specifies the snapshot name.

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-1'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo/my-snapshot-1/_restore'
url = host + path

r = requests.post(url, auth=awsauth)

print(r.text)

$ python restore.py
{"accepted":true}

restore-one.py

Restore specific index. Here we are specifying a specific index with payload.

import requests
from requests_aws4auth import AWS4Auth

AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
region = 'us-west-1'
service = 'es'

awsauth = AWS4Auth(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region, service)

host = 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/'

path = '_snapshot/es-test-index-repo/my-snapshot-1/_restore'
url = host + path

payload = {"indices": "my-index"}

headers = {"Content-Type": "application/json"}

r = requests.post(url, auth=awsauth, json=payload, headers=headers)

print(r.text)

$ python restore-one.py
{"accepted":true}

curl

Restore all indices

curl -XPOST 'elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/my-snapshot-1/_restore'

{"accepted":true}

Restore specific index

$ curl -XPOST 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/_restore' -d '{"indices": "my-index"}' -H 'Content-Type: application/json'

Step7-2 Restore to other ES domain

When restoring to another domain, we need to register the same snapshot repository in the other domain.

Step3: Register a Manual Snapshot repository in the other domain.

※Change host in register-repo.py and execute it.

register-repo.py

$ python register-repo.py
{"acknowledged":true}

Then restore to another domain. Change host and execute it.

Restore done with restore.py

$ python restore-all.py
{"accepted":true}

curl

curl -XPOST 'https://elasticsearch-domain.us-west-2.es.amazonaws.com/_snapshot/es-test-index-repo/my-snapshot-1/_restore'

Conclusion

We illustrated how to restore from a Amazon ES Manual Snapshot. We hope this blog will be helpful when restoring to a different domain or restoring from a snapshot created during maintenance.

References

Working with Amazon Elasticsearch Service Index Snapshots

Share this article

facebook logohatena logotwitter logo

関連記事

技術調査の結果を表にまとめる際のコツについて考えてみた

技術調査の結果を表にまとめる際のコツについて考えてみた

User avatar
nayu.t.s
2023.05.01
ITSM プラットフォームを選定しよう #サービスマネジメント

ITSM プラットフォームを選定しよう #サービスマネジメント

サービスカタログを作成しよう #サービスマネジメント

サービスカタログを作成しよう #サービスマネジメント

私流・データ分析基盤の技術調査のコツを整理してみた

私流・データ分析基盤の技術調査のコツを整理してみた

User avatar
nayu.t.s
2022.09.27
Classmethod's white logo
Facebook's logo
X's logo
YouTube's logo

© Classmethod, Inc. All rights reserved.